Generally, a transactional computer system receives, processes and stores data. The system may also transmit data for further processing. During processing, data stored in memory of the transaction system will be lost if there is a failure that brings the system down, e.g., a processor failure.
A transaction system has four primary performance measures: capacity, latency, data integrity, and availability. As the transaction volume increases, it is critical that a high data integrity objective be maintained since even a very small fraction of data loss can result in the absolute loss of a relatively large number of transactions. However, the additional processing required to attain this goal can have significant impacts on the other performance measures. In one type of transaction system, during the lifetime of each transaction, a series of several events containing raw data related to the transaction is produced by an event generator and received by the transaction system. When the last event is received, a record for the transaction is generated by the transaction system. This record is then forwarded to another system, a record collector, for further processing. Thus, there are three primary stages in the flow of data through the transaction system: event receiving, record formatting, and record forwarding.
Traditionally, two data storage techniques have been used to ensure data integrity. The first is to store records after they have been formatted. However, in this strategy, if a failure occurs, events which have not yet been formatted into records are lost. This method is used to reduce processing time and to conserve disk space since a record is typically much smaller than the total size of all the events related to the record.
As technology has progressed, inexpensive disk storage has become more readily available. A second technique improves upon the first by saving events. However, in this technique, events are stored on disk only after they have been assembled and correlated to their transactions. Thus, in addition to storing the raw events, the structure of the events in relation to the transactions gets stored as well. Such disk-stored structural information is useful to simplify the reformatting of records upon system recovery after a failure. However, the storing of raw events in conjunction with structural information consumes processing resources, and the amount of processor time required by this method can become excessive as transaction volume increases. For example, the same event may be written to disk multiple times. This is likely because for every transaction that has received a new event since the last disk update, all events belonging to the transaction get written to disk to maintain the structural relationship.
In high throughput systems, the processor must be utilized mainly for transaction processing. Data integrity schemes present additional demands on processor utilization which reduce the capacity of the system. To the contrary, these demands are overhead processing costs.